Crowdsourcing Objective Answers to Subjective Questions Online

In this demonstration, we show how Ranker’s algorithms use diverse sampling, measurement, and algorithmic techniques to crowdsource answers to subjective questions in a real-world online environment where user behavior is difficult to control. Ranker receives approximately 8 million visitors each month, as of September 2013, and collects over 1.5 million monthly user opinions. Tradeoffs between computational complexity, projected user engagement, and accuracy are required in such an environment, and aggregating across diverse techniques allows us to mitigate the sizable errors specific to individual imperfect crowdsourcing methods. We will specifically show how relatively unstructured crowdsourcing can yield surprisingly accurate predictions of movie box-office revenue, celebrity mortality, and retail pizza topping sales. Online Crowdsourcing Opinion aggregation is a billion dollar online business that includes both business-to-business market research and direct to consumer websites such as TripAdvisor, which aggregates opinions about hotels, and Yelp, which aggregates opinions about local businesses. Unlike many opinion aggregators, Ranker’s platform is not domain specific and encompasses opinions about entertainment (“most anticipated movies”), sports (“best nba players”), books (“books that changed my life”), the future (“celebrity death pool”), and many other domains. As interest in aggregated opinions grows (Iyer, 2013a), traffic to Ranker’s lists have grown as well, giving Ranker a steady stream of consumer opinions about a variety of topics. The Wisdom of Ranker Crowds Ranker’s algorithm to aggregate user opinions is based on the Wisdom of Crowds principle of aggregating as many diverse sources of “signal” as possible (Larrick, Mannes, & Soll, 2010), in the hopes that the error specific to any individual source of variance cancels out when aggregated. Visitors to Ranker provide their opinions in several different ways. The most common interaction is for visitors to vote on particular items on a list, ostensibly based on whether the item belongs on the list (e.g. is The Godfather one of the best movies of all time?), but our analyses have shown that items do get downvoted more as they move higher on lists, indicating that votes are based both on whether an item belongs on a list and on whether an item belongs at the specific position on a list. Since a registration is not required for voting, there are relatively low barriers to engagement, which enables Ranker to achieve 10-15% engagement on votable lists with voters voting on 10-15 items. These opinions tend to be “shallow” compared to other inputs, and reflect mass opinion as opposed to the opinions of particularly knowledgeable individuals. The other input that Ranker collects from users comes in the form of “reranked” user lists where a user makes their own version of a particular list. These lists may be created from scratch or may be based on existing lists, with the order changed. The barrier to entry is higher, and so these lists tend to be made by individuals who have greater knowledge of the domain. As such, while we receive far fewer user lists, compared to user votes, we weigh user lists heavily in our algorithms in an attempt to reach a balance between expert and non-expert opinions, both of which have some amount of uncorrelated error. Further, we consider both the list position of an item, which indicates the level of passion that a user has for an item, and the number of times an item appears on user lists, which is indicative of the popularity that an item has among knowledgeable individuals. In keeping with the principle that diversity of measurement leads to better results, we feel that the combination of “shallow” voting behavior, “expert” popularity, and “expert” passion yields better algorithmic results than any particular single measure of user opinions. 93 Human Computation and Crowdsourcing: Works in Progress and Demonstration Abstracts AAAI Technical Report CR-13-01